W11. Positive Definite Optimization

Author

Salman Ahmadi-Asl

Published

April 3, 2026

1. Theory

1.1 Introduction and Motivation

In one dimension, the question “is this number positive?” has a clear answer: if and only if for every nonzero . Positive definite matrices are the direct matrix analogue of positive numbers. A symmetric matrix is called positive definite if the quadratic expression is positive for every nonzero vector . This single condition — sometimes called the positive-definite inequality — unlocks a rich theory that connects eigenvalues, determinants, matrix factorizations, function shapes, and optimization, all at once.

Positive definite matrices appear throughout applied mathematics and engineering:

  • Geometry: The Hessian of a strictly convex function is positive definite; its level sets are ellipsoids.
  • Optimization: A critical point where the Hessian is positive definite is guaranteed to be a local minimum.
  • Statistics: Covariance matrices are positive semi-definite; valid correlation structures require this.
  • Numerical methods: Cholesky factorization — a fast, numerically stable solver — exists exactly when the matrix is positive definite.
  • Machine learning: Kernel matrices must be positive semi-definite for the theory to be well-posed.

This article builds the complete theory from definition to application, covering the eigenvalue test, Sylvester’s criterion, the Cholesky decomposition, the geometry of quadratic forms, optimization via the Hessian, convexity, and the extension to complex Hermitian matrices.

1.2 Positive Definite and Positive Semi-Definite Matrices

Definition (Positive Definite, PD): A symmetric matrix is called positive definite if

We write .

Definition (Positive Semi-Definite, PSD): A symmetric matrix is called positive semi-definite if

We write . The difference is that a PSD matrix allows equality for some nonzero ; a PD matrix does not.

Two further cases arise in practice. A symmetric matrix is negative definite (ND) if for all , i.e.  is PD. It is indefinite if takes both positive and negative values for different nonzero vectors. In the indefinite case the matrix is neither PD, PSD, ND, nor NSD.

The expression is called a quadratic form; it is the subject of Section 1.3.

1.2.1 Simple Examples

A diagonal matrix has . This sum is positive for all if and only if every . Thus:

  • A diagonal matrix is PD iff all diagonal entries are positive.
  • It is PSD iff all diagonal entries are nonneg­ative.

For a symmetric matrix , the quadratic form . Completing the square in :

For this to be positive for all , both and (i.e. ) are necessary and sufficient. This is Sylvester’s criterion for the case.

1.3 Quadratic Forms
1.3.1 Definition and Matrix Representation

A quadratic form in variables is a homogeneous polynomial of degree 2:

Every quadratic form can be written as where is a symmetric matrix. The symmetry is enforced by splitting each coefficient: if the original coefficient of (with ) is , then ; diagonal entries equal the coefficient of directly.

Rule for reading off : Given ,

  • (coefficient of ),
  • (half the coefficient of ).

Example: . The coefficient of is , so . Thus .

Symmetry matters because it gives:

  • a unique matrix representation,
  • real eigenvalues (Spectral Theorem),
  • a direct connection between the sign of and the eigenvalues of .
1.3.2 Diagonalizing Quadratic Forms

The Spectral Theorem states that every real symmetric matrix has an orthogonal eigendecomposition with orthogonal and . Under the change of variables (equivalently ):

The quadratic form becomes a pure sum of squares in the new coordinates. Since is invertible, . Therefore:

This equivalence is the foundation of the eigenvalue test.

1.4 Equivalent Conditions for Positive Definiteness

For a symmetric matrix , the following are equivalent:

  1. Definition: for all .
  2. Eigenvalue test: All eigenvalues .
  3. Sylvester’s criterion: All leading principal minors are positive, i.e.  for , where is the top-left submatrix of .
  4. Cholesky factorization: There exists an invertible lower-triangular matrix with positive diagonal such that .

Each condition provides a different computational pathway.

1.4.1 Eigenvalue Test

The eigenvalue test follows from the diagonalization argument in Section 1.3.2. It is the most conceptually transparent but computationally expensive for large matrices (finding eigenvalues requires solving the characteristic polynomial).

For symmetric PSD: All eigenvalues (the inequality is not strict; some eigenvalue may be zero).

1.4.2 Sylvester’s Criterion

The leading principal minor of order is . Sylvester’s criterion says:

For a matrix , this reduces to and . This is fast to check and does not require computing eigenvalues.

Important note on PSD: For positive semi-definiteness, Sylvester’s criterion does not simply become . A matrix can have all non-negative leading principal minors yet fail to be PSD (it requires checking all principal minors, not just the leading ones). For the purposes of this course, use the eigenvalue test for PSD.

1.4.3 Checking Definiteness at a Glance

A quick workflow for a symmetric matrix :

Condition Classification
and Positive definite
and Negative definite
Indefinite
Positive or negative semi-definite (check sign of trace)
1.5 The Cholesky Decomposition

Theorem (Cholesky): A symmetric matrix is positive definite if and only if there exists a unique lower-triangular matrix with strictly positive diagonal entries such that .

The factorization is the Cholesky decomposition of . It is the matrix analogue of taking a positive square root of a number. Because is lower-triangular, the decomposition is easy to compute via forward substitution, and it is numerically very stable — roughly twice as fast as LU decomposition.

Why it implies PD: For any , if is invertible then , so .

LDL^T factorization: A closely related form is where is unit lower-triangular (diagonal 1’s) and . This is obtained from Gaussian elimination and the pivots are the ’s. The Cholesky form follows when (so ). The form is useful because it directly exhibits the quadratic form as:

The pivots in the factorization are positive if and only if .

1.6 Geometric Shapes of Quadratic Forms

The definiteness of determines the shape of the surface :

Classification Eigenvalue signs Shape of Critical point
Positive definite All Elliptic paraboloid (bowl) Unique global minimum
Positive semi-definite All , some Parabolic cylinder (trough) Minimum along a line
Indefinite Mixed signs Hyperbolic paraboloid (saddle) Saddle point
Negative definite All Inverted paraboloid Unique global maximum

The level sets are:

  • Ellipses (or ellipsoids) when ,
  • Parallel lines (or hyperplanes) when is PSD with a zero eigenvalue,
  • Hyperbolas when is indefinite.

These shapes explain why a PD Hessian at a critical point signals a minimum.

1.7 Optimization: The Hessian and Critical Points

Consider a twice-differentiable function .

  • A point is a critical point if .
  • The Hessian matrix is the symmetric matrix of second partial derivatives: .

Second Derivative Test (multi-variable):

Condition on Nature of
Strict local minimum
Strict local maximum
indefinite Saddle point
singular Test inconclusive

The intuition: the quadratic approximation (the gradient term vanishes at a critical point) is a bowl when , an inverted bowl when , and a saddle when is indefinite.

Standard procedure:

  1. Compute and solve to find all critical points.
  2. Compute at each critical point.
  3. Apply the eigenvalue test or Sylvester’s criterion to classify .
1.8 Convexity

Definition (Convex function): is convex if for all and :

If the inequality is strict for and , then is strictly convex.

Hessian criterion for convexity: For a twice-differentiable function :

  • is convex for all ,
  • is strictly convex for all (the converse is not quite true in general, but this sufficient condition is what we use).

Why convexity matters: For a convex function, every local minimum is automatically a global minimum. This is the cornerstone of convex optimization, which underpins machine learning, signal processing, and operations research.

1.9 Hermitian Positive Definite Matrices

For complex matrices, the role of symmetric matrices is played by Hermitian matrices: is Hermitian if , where denotes the conjugate transpose (). For a Hermitian matrix, for every , so positivity is well-defined.

Definition (HPD): A Hermitian matrix is Hermitian positive definite (HPD) if

Definition (HPSD): is Hermitian positive semi-definite (HPSD) if for all .

Exactly the same equivalences hold as in the real case:

Condition Equivalent statement
is HPD All eigenvalues of are real and positive
is HPD with lower-triangular invertible (Cholesky)
is HPD All leading principal minors of are positive (Sylvester)
is HPSD All eigenvalues of are real and nonneg­ative
is HPSD for some possibly singular

Note that the eigenvalues of a Hermitian matrix are always real (a consequence of the Spectral Theorem for Hermitian matrices), so the condition “eigenvalues are positive” is non-vacuous over .

Construction via : Any matrix of the form with invertible is automatically HPD. This is analogous to how with forces in the scalar case.

Key warning: Positivity is only defined for Hermitian matrices over . For a non-Hermitian complex matrix, may not be real, and the notion of HPD does not apply.

2. Definitions

  • Positive definite (PD) matrix: A symmetric matrix such that for all nonzero .
  • Positive semi-definite (PSD) matrix: A symmetric matrix such that for all .
  • Negative definite (ND) matrix: A symmetric matrix such that for all nonzero (equivalently, is PD).
  • Indefinite matrix: A symmetric matrix that is neither PD, PSD, ND, nor NSD; equivalently, takes both positive and negative values.
  • Quadratic form: A homogeneous polynomial of degree 2 in variables, written as with symmetric.
  • Leading principal minor of order : The determinant of the top-left submatrix of , denoted .
  • Sylvester’s criterion: for all .
  • Cholesky decomposition: The factorization (real) or (complex), where is lower-triangular with positive diagonal entries; exists and is unique when .
  • LDL factorization: The factorization where is unit lower-triangular and is diagonal; the diagonal entries of (pivots) are positive iff .
  • Critical point: A point where .
  • Hessian matrix: The symmetric matrix of second partial derivatives of , denoted ; governs local curvature.
  • Convex function: A function such that for all and .
  • Hermitian matrix: A complex matrix satisfying (conjugate transpose equals itself).
  • Hermitian positive definite (HPD): A Hermitian matrix with for all nonzero complex .
  • Hermitian positive semi-definite (HPSD): A Hermitian matrix with for all complex .

3. Formulas

  • Quadratic form in matrix notation: , where is symmetric.
  • Symmetric matrix from quadratic form: $A_{ii} = $ coefficient of ; for .
  • Diagonalized quadratic form: where and is the orthogonal matrix of eigenvectors.
  • Completing the square (): .
  • Sylvester’s criterion (): and .
  • Cholesky decomposition: (real); (complex Hermitian).
  • LDL factorization and quadratic form: where and are the pivots.
  • Eigenvalue product formula: .
  • Second derivative test (classification): at a critical point , local min; local max; indefinite saddle.
  • Hessian convexity criterion: convex for all .
  • Determinant of HPD matrix: (all ).

4. Practice

4.1 Design a Non-Positive-Definite Symmetric Matrix (Lab 9, Task 1)

Given a symmetric matrix with positive coefficients satisfying , construct a specific matrix that is not positive definite.

Click to see the solution

Key Concept: For a symmetric matrix with , positive definiteness requires both and . The condition does not imply ; they are independent constraints.

We need , , but (so that the matrix is not PD).

Construction: Take , , .

  • Check : . ✓
  • Compute .

So satisfies all conditions but is indefinite (not positive definite).

Why does this work? The condition places a constraint on the arithmetic mean of the diagonal entries relative to the off-diagonal, but positive definiteness requires (the geometric mean condition). These two constraints are compatible — does not force — so we can violate PD while satisfying the given constraint.

4.2 Prove is SPD When is SPD (Lab 9, Task 2)

If is symmetric positive definite, show that is also symmetric positive definite.

Click to see the solution

Step 1: Show is symmetric.

Since is symmetric, . Using the identity :

So is symmetric.

Step 2: Show is positive definite.

Since is symmetric, by the Spectral Theorem it has real eigenvalues . Since , all .

The eigenvalues of are . Since each , each .

Therefore all eigenvalues of are positive, which (by the eigenvalue test) means .

Conclusion: is both symmetric and positive definite, so is SPD.

4.3 Matrix of a Quadratic Form and LDL Factorization (Lab 9, Task 3)

The quadratic form is positive for all . Find its symmetric matrix , compute the factorization, and identify the scalars and the linear substitution .

Click to see the solution

Step 1: Expand the quadratic form and read off .

The coefficient of is , the coefficient of is , and the coefficient of is . So , , :

Step 2: Compute via Gaussian elimination.

We eliminate the entry. The multiplier is .

  • Pivot 1: .
  • New entry (Schur complement): .

Therefore:

Verification: ✓

Step 3: Identify the substitution.

Set .

Then , which matches the original expression. ✓

The pivots and confirm that .

4.4 Test for Minimum via Hessian (Lab 9, Task 4)

Determine whether has a local minimum at the point .

Click to see the solution

Step 1: Verify is a critical point.

At : ✓ and ✓.

Step 2: Compute the Hessian at .

At :

Step 3: Classify using Sylvester’s criterion.

  • , but .

Since , the Hessian is indefinite.

Conclusion: is a saddle point, not a local minimum.

4.5 Find and Classify All Critical Points (Lab 9, Task 5)

Find and classify all critical points of .

Click to see the solution

Step 1: Find critical points.

Since the - and -equations are independent, the four critical points are all pairwise combinations: , , , .

Step 2: Compute the Hessian.

This is a diagonal Hessian, so definiteness is determined directly by the signs of the diagonal entries.

Step 3: Classify each critical point.

Point Classification
Local minimum
Saddle point
Saddle point
Local maximum

Summary: The function has one local minimum at , one local maximum at , and two saddle points at and .

4.6 Apply Sylvester’s Criterion to Classify Four Matrices (Lab 9, Task 6)

Given:

Determine which matrix is positive definite (without computing eigenvalues). Then find such that .

Click to see the solution

Apply Sylvester’s criterion to each matrix : check and .

: ; → Indefinite (not PD).

: → Negative definite (not PD; all eigenvalues negative).

: ; → Positive semi-definite (not PD; one zero eigenvalue).

: ; → is positive definite. ✓

Finding with :

Since is indefinite, takes negative values somewhere. We need , which means has one positive and one negative eigenvalue; there exist directions where the quadratic form is negative.

Try :

4.7 Matrix, Pivots, Rank, Eigenvalues, and Determinant of a Degenerate Form (Lab 9, Task 7)

For the quadratic form , find the matrix , its pivots, rank, eigenvalues, and determinant.

Click to see the solution

Step 1: Write as an outer product.

Let . Then . So .

Step 2: Rank.

All rows of are scalar multiples of , so .

Step 3: Eigenvalues.

For a rank-1 matrix , the eigenvalues are:

  • (corresponding eigenvector ),
  • (the two-dimensional null space is perpendicular to ).

Step 4: Pivots.

Performing row reduction on : multiply row 1 by and add to row 2 to get zero; multiply row 1 by and subtract from row 3 to get zero. All three operations reveal that only the first pivot is nonzero, , and (the subsequent rows become all zeros).

Step 5: Determinant.

This is consistent with rank 1: a singular matrix has determinant 0.

4.8 Check Convexity via Hessian (Lab 9, Task 8)

Check the convexity of .

Click to see the solution

Key Concept: A twice-differentiable function is convex iff its Hessian is positive semi-definite everywhere. Since is a polynomial of degree 2, its Hessian is constant (independent of ).

Compute the Hessian:

Check positive definiteness using Sylvester’s criterion:

  • . ✓
  • . ✓

Since (which implies ), the Hessian is positive definite everywhere. Therefore is strictly convex on .

4.9 Check Hermitian Positive Definiteness (Lab 9, Task 9)

Determine whether is Hermitian positive definite.

Click to see the solution

Step 1: Verify is Hermitian ().

We need . Here , so . ✓

The diagonal entries are real: , . ✓

So is Hermitian.

Step 2: Apply Sylvester’s criterion.

For a Hermitian matrix, the criterion is and .

  • . ✓
  • . ✓

(Note: because .)

Conclusion: is Hermitian positive definite (HPD).

4.10 Classify Five Matrices (Assignment 9, Task 1)

Determine whether each of the following matrices is positive definite, positive semi-definite, or neither.

  1. , (b) , (c) , (d) , (e) .
Click to see the solution

Key tool: For each symmetric matrix, check and .

(a) :

Diagonal matrix with entries and . Positive definite.

(b) :

, . Indefinite (neither PD nor PSD).

(c) :

Diagonal with entries and : eigenvalues are and . Positive semi-definite (not PD).

(d) :

, . Positive definite.

(e) :

Diagonal with entries and . Negative definite (neither PD nor PSD).

4.11 Find All That Make a Matrix Positive Definite (Assignment 9, Task 2)

For what values of is positive definite?

Click to see the solution

Apply Sylvester’s criterion:

  • . ✓ (satisfied for all ).
  • .

Conclusion: is positive definite if and only if .

4.12 Find All That Make a Matrix Positive Definite (Assignment 9, Task 3)

For what values of is positive definite?

Click to see the solution

Apply Sylvester’s criterion:

  • .
  • or .

Combining both conditions (we need AND ): .

Conclusion: if and only if .

4.13 Sylvester’s Criterion for a Matrix (Assignment 9, Task 4)

Use Sylvester’s criterion to determine whether is positive definite.

Click to see the solution

Compute the three leading principal minors.

:

:

: Expand along the first row:

All three leading principal minors are positive. By Sylvester’s criterion, is positive definite.

4.14 Eigenvalues and Definiteness of a Matrix (Assignment 9, Task 5)

Find the eigenvalues of and determine whether is positive definite.

Click to see the solution

Step 1: Find the characteristic polynomial.

Step 2: Solve using the quadratic formula.

So and .

Step 3: Apply the eigenvalue test.

Both eigenvalues are positive (since , so ).

Conclusion: is positive definite.

4.15 Symmetric Matrix of a Quadratic Form in Three Variables (Assignment 9, Task 6)

Consider .

  1. Write as with symmetric.
  2. Determine whether is positive definite.
Click to see the solution

(a) Reading off the symmetric matrix.

Term Coefficient Matrix entry

(b) Sylvester’s criterion.

  • . ✓
  • . ✓
  • . Expand along the first row:

All three leading principal minors are positive, so is positive definite.

4.16 Complete the Square to Verify Positive Definiteness (Assignment 9, Task 7)

For :

  1. Write the associated symmetric matrix.
  2. Complete the square to express as a sum of squares.
  3. Determine whether is positive definite.
Click to see the solution

(a) Symmetric matrix.

The coefficient of is 2, of is 2, and of is 2 so .

(b) Complete the square.

Factor out 2 from the -terms:

So .

(c) Positive definiteness.

Both coefficients ( and ) are positive, and the expression is a sum of two squares in terms of independent linear combinations. Specifically:

  • If : the second term , so .
  • If and : the first term , so .

Therefore for all : is positive definite.

4.17 Prove Sylvester’s Criterion for Matrices (Assignment 9, Task 8)

Let be a symmetric matrix. Prove that is positive definite if and only if and .

Click to see the solution

We prove both directions.

() Assume .

  • Take : . So .
  • Since , the quadratic with satisfies for all real . A quadratic with that is positive everywhere has negative discriminant: , i.e. , i.e. .

() Assume and .

Complete the square:

Both coefficients are positive ( and ). For any :

  • If : the second term , so .
  • If , : the first term , so .

Thus .

4.18 Find and Classify a Critical Point (Assignment 9, Task 9)

Find and classify the critical points of .

Click to see the solution

Step 1: Solve .

From (1): . Substitute into (2): . Then .

Critical point: .

Step 2: Compute and classify the Hessian.

  • , .

, so is a local minimum.

4.19 Analyze Critical Points of a Quadratic Function (Assignment 9, Task 10)

For :

  1. Find all critical points.
  2. Compute the Hessian matrix.
  3. Classify the critical point.
Click to see the solution

(a) Find critical points.

Substitute: , .

Critical point: .

(b) Hessian.

(c) Classification.

  • , .

, so is a local minimum (and since is a positive definite quadratic, it is in fact the global minimum).

4.20 Find and Classify Critical Points of a Cubic Function (Assignment 9, Task 11)

Find and classify all critical points of . (There are four critical points.)

Click to see the solution

This is identical to Lab 9, Task 5. See the solution in 4.5 above for the complete derivation.

Summary of results:

Critical Point Classification
Local minimum
Saddle point
Saddle point
Local maximum
4.21 Degenerate Critical Point Analysis (Assignment 9, Task 12)

Consider .

  1. Compute the Hessian at .
  2. Use the Hessian to analyze the nature of .
  3. Is actually a local minimum? Justify.
Click to see the solution

First, verify is a critical point:

(a) Hessian at .

At :

(b) Analysis via Hessian.

. The Hessian is indefinite, so the second-order test is inconclusive (it cannot classify ).

(c) Is a local minimum?

Evaluate along the line :

At , . For small : the dominant term is , so .

Therefore decreases along near , and is not a local minimum. It is a saddle point (the second-order test failed because the Hessian was singular and indefinite).

4.22 Prove Convexity of an Exponential Function (Assignment 9, Task 13)

Show that is convex on .

Click to see the solution

Strategy: Show that the Hessian for all .

Step 1: Compute the Hessian.

Let . Then .

Step 2: Check positive semi-definiteness of .

Since , it suffices to check .

  • always.
  • always.

By Sylvester’s criterion, for all . Therefore everywhere.

Conclusion: is strictly convex on .

4.23 Determine Convexity of a Quadratic Function (Assignment 9, Task 14)

Determine whether is convex.

Click to see the solution

This is identical to Lab 9, Task 8. See the solution in 4.8 above.

Conclusion: is PD everywhere (determinant 7 > 0, ). Therefore is strictly convex.

4.24 Prove Convexity Iff Hessian Is PSD (Assignment 9, Task 15)

Let where is symmetric. Prove that is convex if and only if .

Click to see the solution

Step 1: Compute the Hessian of .

The gradient is , and the Hessian is:

Step 2: Apply the Hessian convexity criterion.

A twice-differentiable function is convex if and only if its Hessian is positive semi-definite at every point. Since is constant:

Remark: If , then is strictly convex and has a unique global minimizer .

4.25 Maximize a Profit Function (Assignment 9, Task 16)

A company’s profit is .

  1. Find the production levels that maximize profit.
  2. Verify this is a maximum.
Click to see the solution

(a) Find the critical point.

From (1): . Substitute into (2): . Then .

Critical point: .

(b) Verify it is a maximum.

  • .
  • .

Since and , the Hessian is negative definite. Therefore is a local maximum (and, since is a concave quadratic, also the global maximum).

4.26 Prove for SPD Matrices (Assignment 9, Task 17)

Let be an symmetric positive definite matrix. Prove that .

Click to see the solution

Since is symmetric, the Spectral Theorem guarantees that has real eigenvalues (counted with multiplicity).

Since , the eigenvalue test states that all .

The determinant of a matrix equals the product of its eigenvalues:

Since each , the product is a product of positive numbers, hence positive:

4.27 Prove Positive-Definite Inequality from Eigenvalue Condition (Assignment 9, Task 18)

Suppose is a symmetric matrix with all eigenvalues positive. Prove that for all .

Click to see the solution

Since is symmetric, by the Spectral Theorem there exists an orthogonal matrix such that , where contains the eigenvalues with all .

Step 1: Change coordinates. Let . Since is orthogonal (hence invertible), .

Step 2: Expand the quadratic form.

Step 3: Bound from below. Since , at least one component . Therefore at least the -th term in the sum satisfies (as and ). All other terms are nonneg­ative. Thus:

4.28 Prove the Sum of Two PD Matrices Is PD (Assignment 9, Task 19)

Let and be two symmetric positive definite matrices. Prove that is also positive definite.

Click to see the solution

We need to show for all .

Expand:

Since , we have for all .

Since , we have for all .

Therefore, for any :

Symmetry: is symmetric because .

Hence is symmetric positive definite.

4.29 Classify a General Quadratic Function (Assignment 9, Task 20)

Consider with .

  1. Find the condition on for to have a local minimum at .
  2. Find the condition for to be a local maximum.
  3. Find the condition for to be a saddle point.
Click to see the solution

Identify the Hessian. Since is a pure quadratic (degree 2 with no constant or linear terms), is the only critical point, and the Hessian of is the constant matrix:

Equivalently, the classification of is determined by the definiteness of , which has and .

(a) Local minimum: and .

Since is already given, the condition is .

(b) Local maximum: requires , i.e. . But the problem specifies , so the first leading principal minor of is positive. A negative definite matrix requires all leading principal minors to alternate in sign starting negative, so already contradicts . No local maximum is possible when .

(c) Saddle point: is indefinite, i.e. , i.e. .

4.30 Verify Positive Definiteness by Direct Computation (Lecture 9, Example 1)

Let . Verify directly from the definition that is positive definite.

Click to see the solution

Take any nonzero vector .

Since , at least one of is nonzero. If , then ; if , then . In either case .

Conclusion: is positive definite by definition.

4.31 Diagonalize a Quadratic Form and Verify Positivity (Lecture 9, Example 2)

For , determine whether is always positive using the eigenvalue test, and write in diagonal form.

Click to see the solution

Step 1: Write the symmetric matrix.

.

Step 2: Find eigenvalues.

Eigenvalues: , .

Step 3: Eigenvalue test.

Both eigenvalues are positive, so and for all .

Step 4: Diagonal form.

After the orthogonal change of variables (where is the matrix of normalized eigenvectors):

This is manifestly a sum of positive terms, confirming the quadratic form is always positive.

4.32 Full Optimization Workflow (Lecture 9, Example 3)

Let . Find all critical points and classify them.

Click to see the solution

Step 1: Compute the gradient and solve .

Subtract (2) from (1): . Then .

Critical point: .

Step 2: Compute the Hessian.

Step 3: Apply Sylvester’s criterion.

  • , .

, so is a local minimum.

Minimum value: .

4.33 Cholesky Factorization (Lecture 9, Example 4)

Compute the Cholesky decomposition for .

Click to see the solution

Step 1: Verify .

, . So and a Cholesky factorization exists.

Step 2: Find .

Expand :

  • : (positive diagonal required).
  • : .
  • : .

Verification:

4.34. Compute the Singular Value Decomposition (Test II Recap, Task 1)

Find the Singular Value Decomposition for the matrix

Click to see the solution

Key Concept: To find the SVD of an matrix , compute the eigenvalues of (an matrix) to get the singular values, derive from the eigenvectors of , derive from the eigenvectors of (an matrix), and arrange both in descending order of singular values.

Step 1: Compute and .

Step 2: Find the eigenvalues.

For (already diagonal): , .

For (already diagonal): , , .

The nonzero eigenvalues of and coincide, as guaranteed by theory.

Step 3: Compute singular values.

Order from largest to smallest:

Step 4: Construct .

has the same dimensions as (), with singular values on the main diagonal:

Step 5: Find eigenvectors of to build .

Since is diagonal, the eigenvectors are the standard basis vectors. Matching the descending order :

Step 6: Find eigenvectors of to build .

Since is diagonal, eigenvectors are again standard basis vectors. Matching the order :

Step 7: Verify .

Answer:

4.35. Find Rank, Singular Values, and Fundamental Subspace Dimensions (Test II Recap, Task 2)

For the matrix , find , the singular values of , and the dimensions of the four fundamental subspaces , , , .

Click to see the solution

Key Concept: The rank equals the number of nonzero singular values. For an matrix of rank : , , , .

Here , .

Step 1: Compute .

Step 2: Find eigenvalues of .

Numerically: , so and .

Both eigenvalues are strictly positive, so both are nonzero.

Step 3: Compute singular values.

Step 4: Determine rank.

The number of nonzero singular values equals the rank:

Step 5: Dimensions of the four fundamental subspaces.

Using the rank-nullity theorem and the SVD structure:

Subspace Formula Value

Verification via rank-nullity: ✓ and ✓.

Answer: ; singular values and ; , , , .

4.36. State Theoretical Properties of SVD (Test II Recap, Task 3)

Let with .

(a) How many nonzero singular values does have?

(b) What is the relationship between the eigenvalue spectra of and ?

(c) What are the dimensions of the matrix in the SVD ?

Click to see the solution

Key Concept: The SVD exists for every real matrix. Its theoretical properties connect the rank, eigenvalue spectra, and subspace dimensions of a matrix.

(a) Number of nonzero singular values.

The singular values of are the nonnegative square roots of the eigenvalues of (equivalently, of ). The number of nonzero singular values equals the rank of :

This follows because , so has exactly positive eigenvalues and zero eigenvalues.

(b) Relationship between spectra of and .

is and is . They share the same nonzero eigenvalues. If , the extra eigenvalues of are all zero:

More generally, the nonzero parts of their spectra are identical regardless of which is larger:

This is why computing SVD from the smaller of the two matrices ( or ) is always valid.

(c) Dimensions of .

In the full SVD , the matrix has exactly the same dimensions as :

It has the nonzero singular values on its main diagonal and zeros everywhere else.

Answer: (a) exactly nonzero singular values; (b) and share the same nonzero eigenvalues, with the larger matrix having additional zero eigenvalues; (c) .

4.37. Prove Positive Definiteness via Sylvester’s Criterion (Test II Recap, Task 4)

Prove that the matrix is positive definite.

Click to see the solution

Key Concept: A symmetric matrix is positive definite if and only if all its leading principal minors are strictly positive (Sylvester’s criterion). The -th leading principal minor is the determinant of the top-left submatrix.

Step 1: (top-left minor).

Step 2: (top-left minor).

Step 3: (full determinant).

Expand along the third row:

where the cofactors are:

Therefore:

Conclusion.

All three leading principal minors are strictly positive: , , .

By Sylvester’s criterion, .

4.38. Analyze a Quadratic Form (Test II Recap, Task 5)

Let .

(a) Write in the form and identify the symmetric matrix .

(b) Express as a sum of squares.

(c) Determine whether is positive definite, negative definite, or indefinite.

Click to see the solution

Key Concept: Every quadratic form corresponds to a symmetric matrix with , , and . Positive definiteness is confirmed when all leading principal minors are positive.

(a) Matrix representation.

The coefficient of is , the coefficient of is , and the coefficient of is , so :

(b) Sum of squares by completing the square.

Group the terms involving and complete the square in :

Since is a sum of two squares with no mixed cancellation, it is clear that for all , with equality only at .

(c) Definiteness via Sylvester’s criterion.

Compute the leading principal minors of :

  • ,
  • .

Both minors are strictly positive, so (positive definite).

This is consistent with the sum-of-squares representation: for all .

Answer: ; ; .

4.39. Find Local Extrema of a Multivariable Function (Test II Recap, Task 6)

Find all local extrema of .

Click to see the solution

Key Concept: A local extremum of a differentiable function occurs at a stationary (critical) point where . The nature of each critical point is determined by the Hessian matrix : if , the point is a local minimum; if , a local maximum; if is indefinite, a saddle point.

Step 1: Find the gradient and solve .

From (1): , i.e. .

Substitute into (2): .

Then .

Critical point: .

Step 2: Compute the Hessian.

The Hessian is the matrix of second-order partial derivatives:

Note that the Hessian is constant (the function is a polynomial of degree 2), so it is the same at every point.

Step 3: Classify the critical point using Sylvester’s criterion.

  • ,
  • .

Since all leading principal minors are strictly positive, .

Step 4: Identify the extremum type and value.

Because the Hessian is positive definite, is a local minimum (and in fact a global minimum, since is a strictly convex quadratic).

Answer: has a unique local (and global) minimum at with .

4.40. Identify the Symmetric Matrix of a Quadratic Form and Classify Definiteness (Test II, Task 1)

Given the quadratic form:

(a) Write in the form (find the symmetric matrix ).

(b) Determine whether is positive definite or not.

Click to see the solution

Key Concept: To build the symmetric matrix of a quadratic form, place the coefficient of on the diagonal as , and for each cross term (with ) set . Positive definiteness is then confirmed by Sylvester’s criterion: all leading principal minors must be strictly positive.

(a) Reading off the symmetric matrix.

The coefficients of the squared terms give the diagonal entries directly: , , .

For the cross terms:

  • Coefficient of is , so .
  • Coefficient of is , so .
  • There is no term, so .

Therefore:

(b) Sylvester’s criterion.

Compute the three leading principal minors:

  • .
  • .
  • . Expanding along the first row:

Since all three leading principal minors are strictly positive, is positive definite.

Answer: ; .

4.41. Find Critical Points and Classify via the Hessian (Test II, Task 2)

Given the function:

(a) Find all critical points of .

(b) Compute the Hessian matrix of .

(c) For each critical point, determine its type (local minimum, local maximum, or saddle point).

Click to see the solution

Key Concept: Critical points of a differentiable function satisfy . The Hessian at each critical point reveals the nature of the extremum: if , the point is a local minimum; if , a local maximum; if is indefinite, a saddle point.

(a) Finding the critical point.

Set both partial derivatives to zero:

Substitute into :

The only critical point is .

(b) Hessian matrix.

The second partial derivatives are constant:

(c) Classification at .

Apply Sylvester’s criterion to :

  • ,
  • .

Since , the point is a local minimum. Because is a quadratic with positive-definite Hessian everywhere, it is in fact a global minimum with .

Answer: Unique critical point ; ; is a local (and global) minimum.

4.42. Compute the Full SVD and Scale It (Test II, Task 3)

Given the matrix:

(a) Find the singular values of .

(b) Find the full SVD of , i.e., matrices , , and such that .

(c) Find the SVD of .

Click to see the solution

Key Concept: The singular values of are the square roots of the eigenvalues of (equivalently, of ). The columns of are the unit eigenvectors of ordered by decreasing eigenvalue; the columns of are the unit eigenvectors of in the same order. Scaling by a scalar simply scales all singular values by , leaving and unchanged.

(a) Singular values.

Compute :

The eigenvalues of are and . The singular values (in decreasing order) are:

(b) Full SVD.

Matrix (unit eigenvectors of , ordered by decreasing eigenvalue):

  • : eigenvector , so .
  • : eigenvector , so .

Matrix (same shape as , singular values on the diagonal in decreasing order):

Matrix (unit eigenvectors of ):

Eigenvalues are with eigenvectors:

Verification: . ✓

(c) SVD of .

Scaling by 5 multiplies every singular value by 5 while leaving the orthogonal matrices unchanged:

and are identical to those found in part (b).

Answer: , ; SVD as above; for the singular values become and .

4.43. Prove Properties of Positive Definite Matrices and Find the Definiteness Range (Test II, Task 4)

Assume and are both symmetric positive definite matrices of the same size.

(a) Explain why is also positive definite.

(b) Explain why .

(c) For what values of is the following matrix positive definite?

Click to see the solution

Key Concept: The definition of positive definiteness ( for all ) is linear, so sums of positive definite matrices are positive definite. The determinant of a positive definite matrix is positive because it equals the product of eigenvalues, all of which are positive. Sylvester’s criterion converts the definiteness question for into a system of minor inequalities.

(a) is positive definite.

First, is symmetric: .

For any nonzero vector :

Since , we have ; since , we have . Their sum is therefore also strictly positive. Hence .

(b) .

A real symmetric matrix is positive definite if and only if all its eigenvalues are strictly positive (this follows from the Spectral Theorem). The determinant equals the product of all eigenvalues:

Since every , the product is positive, so .

(c) Values of for which .

Apply Sylvester’s criterion: all leading principal minors of must be strictly positive.

  • . ✓ (no condition on from this minor)
  • .
  • . Expanding along the third row (since the entry is 1 and the rest of its row/column are zero):

All three conditions reduce to the single requirement .

Answer: is positive definite for .